Skip to content

OCPBUGS-9037: If mTLS is required and the canary receives a "certificate required" TLS error, consider the canary probe successful#961

Closed
rfredette wants to merge 2 commits intoopenshift:masterfrom
rfredette:ocpbugs-9037-canary-mtls
Closed

OCPBUGS-9037: If mTLS is required and the canary receives a "certificate required" TLS error, consider the canary probe successful#961
rfredette wants to merge 2 commits intoopenshift:masterfrom
rfredette:ocpbugs-9037-canary-mtls

Conversation

@rfredette
Copy link
Contributor

Without an API to provide the canary with a valid client certificate, the canary probes are guaranteed to fail when mTLS is required by the default ingress controller. However, if the failure is determined to be because the canary is missing a client certificate, it's reasonable to assume that the ingress controller is functional.

This change checks specifically for the TLS alert "certificate required", and if the default ingress controller also has mTLS required, the probe function returns nil, indicating a successful probe.

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 10, 2023
@openshift-ci-robot
Copy link
Contributor

@rfredette: This pull request references Jira Issue OCPBUGS-9037, which is invalid:

  • expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Without an API to provide the canary with a valid client certificate, the canary probes are guaranteed to fail when mTLS is required by the default ingress controller. However, if the failure is determined to be because the canary is missing a client certificate, it's reasonable to assume that the ingress controller is functional.

This change checks specifically for the TLS alert "certificate required", and if the default ingress controller also has mTLS required, the probe function returns nil, indicating a successful probe.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested review from Miciah and knobunc July 10, 2023 20:24
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 10, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from rfredette. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2023
…TLS error, consider the canary probe successful.

Without an API to provide the canary with a valid client certificate,
the canary probes are guaranteed to fail when mTLS is required by the
default ingress controller.

However, if the failure is determined to be because the canary is
missing a client certificate, it's reasonable to assume that the ingress
controller is functional. This change checks specifically for the TLS
alert "certificate required", and if the default ingress controller also
has mTLS required, the probe function returns nil, indicating a
successful probe.
@rfredette rfredette force-pushed the ocpbugs-9037-canary-mtls branch from 46b2fbd to 4dc9007 Compare August 21, 2023 21:38
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 22, 2023

@rfredette: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@frobware
Copy link
Contributor

/assign @frobware

// "certificate required", it means the canary lacks the client certificate necessary to complete its
// check. In that case, verify that the router is configured to require mTLS, and if so, assume the
// router is working as intended.
if opErr.Err.Error() == "tls: certificate required" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this message the same in a FIPS-enabled cluster?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This message is the same on FIPS, although as we discussed out-of-band, I need to see if the message is the same in all TLS versions since it was added in TLS 1.3.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we summarise the testing you've already done with different TLS profiles?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, to summarize the testing with TLS profiles:
We only allow users to set the minimum TLS version, not the maximum. The canary checks are between the router and the ingress-operator pods, both of which support TLS 1.3, so regardless of what the minimum is set to, the connection gets negotiated to v1.3, so we get teh expected "certificate required" error.

defer func() {
// Reset client TLS.
if err := kclient.Get(context.TODO(), defaultName, defaultIC); err != nil {
panic(fmt.Errorf("failed to get default ingresscontroller: %v", err))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to panic here. We can just call t.Fatal.

}
defaultIC.Spec.ClientTLS = originalClientTLS
if err := kclient.Update(context.TODO(), defaultIC); err != nil {
panic(fmt.Errorf("Failed to restore default ingress configuration: %w", err))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to panic here. We can just call t.Fatal.

@frobware
Copy link
Contributor

One concern that I have is:

This change checks specifically for the TLS alert "certificate required", and if the default ingress controller also has mTLS required, the probe function returns nil, indicating a successful probe.

is this really enough to assert status-code==200?

@rfredette
Copy link
Contributor Author

Closing in favor of #978

@rfredette rfredette closed this Oct 6, 2023
@openshift-ci-robot
Copy link
Contributor

@rfredette: This pull request references Jira Issue OCPBUGS-9037. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Without an API to provide the canary with a valid client certificate, the canary probes are guaranteed to fail when mTLS is required by the default ingress controller. However, if the failure is determined to be because the canary is missing a client certificate, it's reasonable to assume that the ingress controller is functional.

This change checks specifically for the TLS alert "certificate required", and if the default ingress controller also has mTLS required, the probe function returns nil, indicating a successful probe.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants